Iteration Aware Prefetching for Large Multidimensional Scientific Datasets

Most caching and prefetching research does not take advantage of prior knowledge of access patterns, or does not adequately address the storage issues inherent with multidimensional scientific data. Armed with an access pattern specified as an iteration over a multidimensional array stored in a disk file, we use prefetching to greatly reduce the number of disk accesses and partially hide the cost of read latency from the user. We assume the pattern of access is not known until runtime, in contrast to chunking methods which preprocess a file for a particular access pattern. Our results on plain unpreprocessed files are competitive with chunking. We can also apply our method to chunked files and achieve additional performance improvements.